Foreground / background separation in digital images

ABSTRACT

A method for providing improved foreground/background separation in a digital image of a scene is disclosed. The method comprises providing a first map comprising one or more regions provisionally defined as one of foreground or background within the digital image; and providing a subject profile corresponding to a region of interest of the digital image. The provisionally defined regions are compared with the subject profile to determine if any of the regions intersect with the profile region. The definition of one or more of the regions in the map is changed based on the comparison.

PRIORITY

This application is a Continuation of U.S. patent application Ser. No.11/744,020, filed May 3, 2007, now U.S. Pat. No. 8,363,908; which claimsthe benefit of priority under 35 USC §119 to U.S. provisional patentapplication No. 60/746,363, filed May 3, 2006.

BACKGROUND OF THE INVENTION

The present invention provides a method and apparatus for providingimproved foreground/background separation in a digital image.

A focus map may be built using a depth from defocus (DFD) algorithm, forexample, as disclosed in “Rational Filters for Passive Depth fromDefocus” by Masahiro Watanabe and Shree K. Nayar (1995), herebyincorporated by reference. The basic idea is that a depth map of a givenscene can be theoretically computed from two images of the same scene.Ideally, for calculating a DFD map, a telecentric lens is used, and onlyfocus varies between the two image acquisitions. This is generally nottrue of existing digital cameras.

Another technique for separating foreground from background is disclosedin US published patent application no. 2006/0285754, which is assignedto the same assignee as the present application, and is herebyincorporated by reference. Here, the difference in exposure levelsbetween flash and non-flash images of a scene are used to provide aforeground/background map. The main advantage of using depth fromdefocus over a flash/non-flash based technique, is that depth fromdefocus is independent of the scene illumination and so can beadvantageous for outdoor or well-illuminated scenes.

A further technique for separating foreground from background isdisclosed in U.S. patent application No. 60/773,714 and Ser. No.11/573,713, which are hereby incorporated by reference. Here, adifference in high frequency coefficients between corresponding regionsof images of a scene taken at different focal lengths are used toprovide a foreground/background map. Again in this case, theforeground/background map is independent of the scene illumination andso this technique can be useful for outdoor or well-illuminated scenes.

In any case, the foreground/background map produced by each of the abovetechniques or indeed any other technique may not work correctly. It isthus desired to provide improved methods of foreground/backgroundseparation in a digital image.

SUMMARY OF THE INVENTION

A method is provided for providing foreground/background separation in adigital image of a scene. A first map is provided including one or moreregions within a main digital image. Each region has one or more pixelswith a common characteristic. A subject profile is providedcorresponding to a region of interest of the main digital image. One ormore of the regions is/are compared with the subject profile todetermine if any of them intersect with the profile region. One or moreof the regions are designated as a foreground region based on thecomparison.

The providing of the first map may include provisionally defining eachregion of the image as foreground or background. The one or more regionsinclude at least one region provisionally defined as foreground.

The designating may include comparing a foreground region with thesubject profile. Responsive to the foreground region not substantiallyintersecting the subject profile, a designation of said foregroundregion is changed to a background region.

The providing of the first map may be based on a comparison of two ormore images nominally of the same scene. One or more of the images thatare compared may include a lower resolution version of the main image.One or more of the images that are compared may include the main digitalimage. Two or more images that are compared may be aligned and/or may bematched in resolution. One or more of the images that are compared maybe captured just before or after the main digital image is captured.

The providing of said first map may include providing two of more imageseach of different focus and nominally of the scene. The method mayinclude calculating from the images a depth of focus map indicatingpixels of the main digital image as either foreground or background. Thefocus map may be blurred. The method may include thresholding theblurred map to an intermediate focus map indicating regions as eitherforeground or background. Regions within said intermediate focus map maybe filled to provide the first map.

The providing of the first map may include providing two or more imageseach of different focus and nominally of the scene. High frequencycoefficients of corresponding regions in the images may be compared todetermine whether the regions are foreground or background to providethe first map.

The providing of the first map may include providing two or more imagesat different exposure levels nominally of the scene. Luminance levels ofcorresponding regions in the images may be compared to determine whetherthe regions are foreground or background to provide the first map.

Any of the methods described herein may be operable in a digital imageacquisition device that is arranged to select the subject profileaccording to content of the main digital image and/or the device may bearranged to operate in a portrait mode wherein the subject profileincludes an outline of a person. The outline may include one of a numberof user selectable outlines and/or may be automatically selected frommultiple outlines based on the content of the main digital image.

Any of the methods described herein may be operable in a general purposecomputer arranged to receive the first map in association with the maindigital image, and/or may be arranged to receive one or more additionalimages nominally of the scene in association with the main digital imageand/or may be arranged to calculate the first map from a combination ofone or more additional images and the main digital image.

The providing of a subject profile may include determining at least oneregion of the image including a face. An orientation may be determinedof the face. The subject profile may be defined as including the faceand a respective region below the face in the main image in accordancewith the orientation.

The providing of the first map may also include analyzing at least oneregion of the main digital image in a color space to determine a colordistribution within the regions. The color distribution may havemultiple distinct color peaks. The regions may be segmented based onproximity of a pixel's color to the color peaks.

The comparing may include, for each region intersecting said subjectprofile, calculating a reference reflectance characteristic, and foreach region not intersecting the subject profile, calculating areflectance characteristic. The non-intersecting region reflectancecharacteristic may be compared with the reference reflectancecharacteristic for a region corresponding in color to thenon-intersecting region. A non-intersecting region may be designated asforeground when the non-intersecting region reflectance characteristicis determined to be within a threshold of the reference reflectancecharacteristic.

A second image may be provided nominally of the scene. Reflectancecharacteristics may be calculated as a function of a difference inluminance levels between corresponding regions in the main image and thesecond image.

The main image may be one of a stream of images. The determining of atleast one region of the main image including a face may includedetecting a face in at least one image of the stream acquired prior tothe main image. The face may be tracked through the stream of images todetermine the face region in the main image.

A further method is provided for foreground/background separation in adigital image of a scene. A first map is provided including one or moreregions provisionally defined as one of foreground or background withina main digital image. One or more of the regions may be analyzed todetermine a distribution of luminance within pixels of the region.Responsive to the luminance distribution for a region having more thanone distinct luminance peak, the region is divided into more than onesub-region based on proximity of pixel luminances to the luminancepeaks. The method further includes changing in the map a designation ofone or more sub-regions based on the division.

The method may include providing a subject profile corresponding to aregion of interest of the main digital image. At least one provisionallydefined region may be compared with the subject profile to determinewhether the region intersects with the profile region. The method mayfurther include changing in the map a designation of one or more regionsor sub-regions based on comparison.

The providing of the first map includes analyzing one or more regions ofthe digital image in a color space to determine a color distributionwithin the region. The color distribution may have multiple distinctcolor peaks. The regions may be segmented based on proximity of pixelcolor to the color peaks. The analyzed regions may be provisionallydefined as foreground within the first map. The digital image may beprovided in LAB space, and the color space may include [a,b] values forpixels and the luminance may include L values for pixels.

A further method is provided for improved foreground/backgroundseparation in a digital image of a scene. A main digital image may beacquired. At least one region of said main image is determined toinclude a face, and an orientation of the face is determined. Aforeground region is defined in the image including the face, and arespective region below the face is also defined in accordance with theorientation.

An apparatus is provided for providing improved foreground/backgroundseparation in a digital image of a scene. The apparatus includes aprocessor and one or more processor-readable media for programming theprocessor to control the apparatus to perform any of the methodsdescribed herein above or below

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the invention will now be described with reference to theaccompanying drawings, in which:

FIG. 1( a) shows an in-focus image of a subject;

FIG. 1( b) shows a DFD map for the image; and

FIG. 1( c) shows the DFD map of FIG. 1( b) partially processed accordingto a preferred embodiment of the invention;

FIG. 2 shows a flow diagram of a method for improvingforeground/background separation according to the preferred embodimentof the invention;

FIG. 3( a) shows a first color segmented version of the foregroundregions of the image of FIG. 1( c);

FIG. 3( b) shows a profile for a subject;

FIG. 3( c) shows the result of combining the profile of FIG. 3( b) withthe regions of FIG. 3( a) according to an embodiment of the presentinvention; and

FIG. 3( d) shows the image information for the identified foregroundregions of the image of FIG. 3( c);

FIG. 4( a) shows another in-focus image of a subject;

FIG. 4( b) shows a DFD map of the image;

FIG. 4( c) shows a first color segmented version of the foregroundregions of the image; and

FIG. 4( d) shows the result of combining a profile with the regions ofFIG. 4( c) according to an embodiment of the present invention;

FIG. 5( a) shows another in-focus image of a subject;

FIG. 5( b) shows a first color segmented version of the foregroundregions of the image; and

FIG. 5( c) shows a further improved color segmented version of theforeground regions of the image when processed according to anembodiment of the present invention;

FIG. 6( a)-(c) show luminance histograms for regions identified in FIG.5( a);

FIG. 7 is a flow diagram illustrating an alternative segmentation methodfor foreground-background separation in digital images;

FIG. 8 shows regions identified with an image processed in accordancewith the method of FIG. 7; and

FIG. 9 is a flow diagram illustrating a further alternative segmentationmethod for foreground-background separation in digital images.

DETAILED DESCRIPTION OF THE EMBODIMENTS

The present invention is employed where there is a need forforeground/background segmentation of a digital image. There are manyreasons for needing to do so, but in particular, this is useful whereone of the foreground or the background of an image needs to bepost-processed separately from the other of the foreground orbackground. For example, for red-eye detection and correction, it can becomputationally more efficient to only search and/or correct red-eyedefects in foreground regions rather than across a complete image.Alternatively, it may be desirable to apply blur only to backgroundregions of an image. Thus, the more effectively foreground can beseparated from background, the better the results of imagepost-processing.

In the preferred embodiment, improved foreground/background segmentationis implemented within digital camera image processing software, hardwareor firmware. The segmentation can be performed at image acquisitiontime; in a background process, which runs during camera idle time; or inresponse to user interaction with image post-processing software. Itwill nonetheless be seen that the invention could equally be implementedoff-line within image processing software running on a general-purposecomputer.

In any case, in the preferred embodiment, a user operating a cameraselects, for example, a portrait mode and optionally a particular typeof portrait mode, for example, close-up, mid-shot, full length or group.In portrait mode, the camera then acquires a main image or indeed thecamera acquires one of a sequence of preview or post-view imagesgenerally of the main image scene. Generally speaking, these preview andpost-view images are of a lower resolution than the main image. Asoutlined above, at some time after image acquisition, image processingsoftware calculates either for the main image or one of thepreview/post-view images an initial foreground/background map.

The preferred embodiment will be described in terms of the initial mapbeing a DFD map, although it will be appreciated that the invention isapplicable to any form of initial foreground/background map as outlinedabove. In the embodiment, the segmentation process provides from theinitial map, a final foreground/background map, where the foregroundregion(s), ideally, contain the image subject and which can be used infurther image processing as required.

FIG. 1( a) shows an in-focus image of a scene including a subject(person) 10 and FIG. 1 (b) the resulting DFD map. The DFD map has, ingeneral, a number of problems in that:

-   -   objects such as the shutters 12 that lie in the neighborhood of        the subject although at different depths appear in-focus (which        is normal, but undesired) and as such can be falsely classified        as foreground objects; and    -   the DFD map is very noisy, i.e., it is far from being smooth.

Referring now to FIG. 2, the foreground/background segmentationprocessing of the DFD map to provide the final foreground/background mapis shown:

The initial DFD map 20, for example, as shown in FIG. 1( b), is firstsmoothed or blurred with a Gaussian kernel, step 22. The DFD map of FIG.1( b) is in a binary form with white regions being classified asforeground and black being background. Smoothing/blurring the map willtend to indicate foreground regions as generally lighter and backgroundregions as generally darker.

A threshold is then applied, step 24, to the smoothed continuouslyvalued image from step 22. This provides a binary map in general havinglarger and smoother contiguous regions than the initial DFD map 20.

Regions of the binary map obtained at step 24 are then filled, step 26,to remove small regions within larger regions. For the initial image ofFIG. 1( a), an initial foreground/background map as shown in FIG. 1( c)is produced. Here foreground is shown as white and background as black.It will be seen that in this map, there is no distinction between theforeground subject region 14 and the region 16 which should be in thebackground.

The pixels classified as background in the image of FIG. 1( c) areexcluded from further processing, step 28, and the remaining regions ofthe images are regarded as provisional foreground regions.

The remainder of the image is segmented by color, using any suitabletechnique, step 30. In the preferred embodiment, a “mean shift”algorithm, based on D. Comaniciu & P. Meer, “Mean Shift: A RobustApproach toward Feature Space Analysis” IEEE Trans. Pattern AnalysisMachine Intell., Vol. 24, No. 5, 603-619, 2002) is employed. In general,this technique involves identifying discrete peaks in color space andsegmenting the image into regions labelled according to their proximityto these peaks.

While this technique can be performed in RGB space, for the sake ofcomputational complexity, the preferred embodiment operates on [a,b]parameters from an LAB space version of the foreground region 14,16pixels. This means that for an image captured in RGB space, only pixelsfor candidate foreground regions need to be transformed into LAB space.In any case, it should be noted that this [a,b] based segmentation isluminance (L in LAB space) independent. This segmentation produces a mapas shown in FIG. 3( a), where the different shaded regions 30(a).30(f)etc represent a region generally of a given [a,b] color combination.

In a first improvement of foreground/background segmentation accordingto the present invention, a portrait template corresponding to theacquired image is provided, FIG. 3( b). The template includes a profile32 of a subject. The exact size of a particular profile can be variedaccording to the focal length for the acquired image in accordance withthe expected size of a subject. It will be seen that while the profile32 shown in FIG. 3( b) is a mid-shot of a subject, the outline can bevaried according to the expected pose of a subject. This can eitherentered manually by a user, by selecting a suitable portrait mode, orpossibly predicted by the image processing software. Thus, the profilemight be a head shot outline or a full body outline, in one of aplurality of poses, or indeed in the case of a group portrait, anoutline of a group.

In any case, the color segments provided in step 30 are combined withthe profile 32 to retain only color regions that overlap to asignificant extent with the profile 32. Thus, with reference to FIG. 3(a), it will be seen that inter alia regions 30(b),(c) and (e) areremoved from the foreground map, while inter alia regions 30(a),(d) and(f) are retained. The final set of foreground regions is shown shaded inFIG. 3( c), with the final background region being indicated as black.It will been seen, however, from FIG. 3( d)) that some regions such assub-region 30(g) of region 30(a) are still not as accurately segmentedas they might be.

It will be seen that sub-regions 30(g)(1) and 30(g)(2), because they mayhave similar [a,b], characteristics have been included in region 30(a)which in turn has been classified as a foreground region, whereassub-region 30(g)(2) should more suitably be classed as a background.

It is also acknowledged that parts of the foreground can be (wrongly)removed from the foreground map from various reasons. For instance, inFIG. 3( d), it can be seen that the subject's right hand has beenremoved from the foreground map because it does not overlap withportrait profile 32.

Another example of the segmentation of steps 22-34 is illustrated withreference to FIG. 4. FIG. 4( a) shows an in-focus image and FIG. 4( b)the DFD map for the image. FIG. 4( c) shows the segmented map aftercolor segmentation, step 30. FIG. 4( d) shows the finalforeground/background map after elimination of regions, such as40(a),(b) that do not overlap significantly to a portrait template 32chosen for the image.

In this case that, because color segmentation did not separate thesubject's hair from the balcony's edges, region 40(c), the balcony edgeshave been wrongly included in the final map as foreground regions.

In a still further example, FIG. 5( a) shows an in-focus image of asubject and FIG. 5( b), the foreground/background map after colorsegmentation, step 30, but before combining the foreground regions witha profile 32. Two segmentation artifacts can be seen at this stage: thesubject's T-shirt 50 and the TV 52 behind are segmented in a singleregion; and, similarly, half the subject's face and hair 54 are mergedinto a single region. The latter defect (accidentally) will not affectthe final results, as both hair and face are ideally included in a finalforeground map. On the contrary, not separating the T-shirt 50 from theTV 52 results in (wrongly) retaining the latter in the foreground map.

In a second improvement of foreground/background segmentation accordingto the present invention, foreground regions are analysed according toluminance, step 36. This step can be performed in addition to,independently of, or before or after step 34. In the preferredembodiment, this analysis is again performed on an LAB space version ofthe foreground region 14, 16 pixels and so can beneficially use only theL values for pixels as is described in more detail below.

In step 36, the intensity of the pixels in regions of the image ofinterest is analysed to determine if the luminance distribution of aregion is unimodal or bimodal. This, in turn, allows difficult images tohave their foreground/background regions better separated by applyingunimodal or bimodal thresholding to different luminance sub-regionswithin regions of the image.

In the case of FIG. 5, both the T-shirt/TV 50/52 and hair/face pairs 54strongly differ in luminance. In step 36, the luminance histogram ofeach segmented foreground region is computed. FIG. 6 shows the luminancehistograms of region #1 comprising the T-shirt/TV 50/52; region #2comprising the hair/face 54; and region #3 shows a typical unimodaldistribution. As can be seen from FIG. 6, the luminance histograms ofregions that should be further segmented (i.e., regions #1 and 2) arebi-modal, whereas others (region #3) are not.

It should also be noted that multi-modal histograms could also be foundfor a region, indicating that the region should be split into more thantwo regions. However, the instances of such a distribution are likely tobe very rare.

Given that regions which exhibit such a bi-modal distribution inluminance should be ideally segmented further, it is useful toconveniently classify a given histogram as either unimodal or bimodal.Referring to FIG. 6, in the preferred embodiment, this classificationcomprises:

-   (i) blurring/smoothing the histogram to reduce artifacts;-   (ii) finding a maximum luminance 60 in the histogram;-   (iii) discarding a given-width interval 62, FIG. 6( a), around the    maximum coordinate (to avoid detection of false maxima);-   (iv) finding the next maximum 64;-   (v) from each of the two maxima, a mode-detection procedure is run    to find the corresponding mode—a Bell shaped distribution around    each maximum, 66, FIG. 6( b);-   (vi-a) if both found modes include a significant portion of the    histogram (i.e., if each spans an interval of luminance levels that    includes more than 20% of the pixels from regions of interest) then    the histogram is declared bimodal, and the minimum value 68 in the    interval between the two maxima is used as a threshold for splitting    the region into 2 sub-regions; otherwise,-   (vi-b) the histogram is said to be unimodal, and nothing is changed.

FIG. 5( c) presents the result of the final segmentation, where one cansee the correct separation of T-shirt/TV and of hair/face pairs. Regionswhich are considered unimodal are not changed.

Using the present invention, more of an in-focus subject can becorrectly separated from the background, even in difficult images, i.e.,images with background located very close to the subject. Even whenportions of background cannot be separated from the foreground or viceversa, the artifacts are less likely to be big, and the final map can bemore useful for further post-processing of the image.

There are a number of practical issues, which need to be considered whenimplementing the invention:

When the initial map is derived from a DFD map, then the scaling factorbetween the in-focus and out-of-focus images will need to be known. Thisneeds to be accessible from the camera configuration at imageacquisition, as it cannot be computed automatically. It is derivablefrom knowing the focal length for the acquired image, and so this shouldbe made available by the camera producer with the acquired image.

It will also be seen that where the initial map is derived from a DFDmap, some shifting between images may have taken place, depending uponthe time between acquiring the two images. It will be seen that thesubject may move significantly with respect to the background, or thewhole scene may be shifted owing to camera displacement. As suchappropriate alignment between images prior to producing the DFD mapshould be performed.

As indicated earlier, the invention can be implemented using either fullresolution images or sub-sampled versions of such images, such aspre-view or post-view images. The latter may in fact be necessary wherea camera producer decides double full resolution image acquisition toprovide a full resolution DFD map is not feasible. Nonetheless, using apair comprising a full-resolution and a preview/postview, or even a pairof previews/postviews for foreground/background mapping may besufficient and also preferable from a computational efficiency point ofview.

It will also be seen that it may not be appropriate to mix flash andnon-flash images of a scene for calculating the DFD map. As such, wherethe main image is acquired with a flash, non-flash preview and post-viewimages may be best used to provide the foreground/background map inspite of the difference in resolution vis-à-vis the main image.

In a still further aspect of the present invention there is provided afurther improved segmentation method for foreground-backgroundseparation in digital images.

In the embodiments of FIGS. 1 to 6, the portrait profile 32 is stored ina database for comparison with color segments for an image.

However, it has been found that an alternative profile can be providedby detecting the position and orientation of one or more faces in animage, and adding to the or each face region, a respective area,preferably including a column below the face region as indicated by theorientation of the face. As before, a profile including each face regionand associated column can be assumed to comprise foreground pixels only.

While this profile can be used instead of the profile(s) 32 of FIGS. 1to 6, the information provided in this presumed foreground column(s),can also be used to provide improved separation of the image intoforeground and background as described in more detail below.

Referring now to FIG. 7, there is provided a flow diagram of thisembodiment.

An image is acquired 700. As before, the image can be either be a pre-or post-view image or include a down-sampled version of a main acquiredimage.

A face is either detected in the acquired image or, if the face isdetected in a previous image of a stream of images including theacquired image, the face region is tracked to determine a face positionand its orientation within the acquired image 710. The detection of theposition and orientation as well as tracking of a detected face ispreferably carried out as disclosed in U.S. patent application No.60/746,363 filed Aug. 11, 2006.

The orientation of a detected/tracked face is used to determine an areabelow the detected face in the direction of the face and the combinedface region and associated area provides a profile template assumed tocontain foreground pixels only, 720.

Referring now to FIG. 8, an acquired image 10 corresponding to the imageof FIG. 5( a) includes a subject and the subject's face is detectedand/or tracked to lie within the region 80. Given the orientation of theface, a region 82 bounding the face region 80 and extending from thebottom of the face region 80 to the edge of the image is defined.

It can be seen that a number of separate objects lie within or intersectthe region 82. In the example, these might comprise regions bounding thesubject's shirt (A), the subject's neck and face right side (B), thesubject's face left side (C) and the subject's hair (D).

Preferably, these objects are segmented, step 730, by means of a colorobject detection technique such as Color Wizard, edge detection, orother region separation techniques applied on color or grey level imagesincluding but not limited to those described for the embodiments ofFIGS. 1 to 6. In the present embodiment, each object A . . . Didentified at step 730 and lying within or intersecting the region 82 isdesignated as being (at least in so far as it intersects the region 82)a foreground object 740.

Preferably, each foreground object that intersects the region 82 isfurther subjected to luminance analysis, step 750, to determine whetherthe luminance distribution of the object is unimodal or bimodal asdescribed above. Applying unimodal or bimodal thresholding to differentluminance sub-objects within objects intersecting the region 82 can leadto better separation of the foreground/background objects. Thus, objectspreviously identified as foreground, may now comprise a sub-objectidentified as a foreground object and a sub-object identified as abackground object.

Again, this analysis is preferably performed on an LAB space version ofthe foreground object pixels and so can beneficially use only the Lvalues for pixels.

Any object (or sub-object) identified in steps 740 and optionally 750that does not lie within or intersect region 82 is designated as abackground object, 760. In this manner, the image is separated intoforeground and background objects.

In this embodiment of the present invention, foreground/backgroundsegmentation is carried out only on a restricted portion of the imageincluding the region 82. In a still further aspect of the presentinvention there is provided a further improved segmentation method forforeground-background separation of complete digital images.

Referring now to FIG. 9, there is provided a flow diagram of thisembodiment.

A first and second image nominally of the same scene are acquired, 900,905. As before, these images can either be pre- or post-view images orinclude a down-sampled version of a main acquired image. For thisembodiment, one of the images is taken with a flash and the otherwithout a flash to provide a difference in exposure levels betweenimages.

The images are aligned (not shown), so that object segments identifiedin the image in steps 900 to 950 (corresponding with steps 700 to step750 of FIG. 7) are assumed to correspond with segments of the imageacquired in step 905.

Where foreground objects identified in step 940/950 are segmented bycolor, each object comprises a number of pixels having a one of a numberof particular color characteristics e.g. similar AB values in LAB space.

The embodiment of FIG. 9 is based on the assumption that each foregroundobject color combination has an associated average reflectivecharacteristic k that represents the expected behavior of any foregroundobject having the same color combination. Therefore, by comparing theaverage reflective characteristic k of any object in the image with theaverage reflective characteristic k of an identified foreground objecthaving the same color combination, the object can be identified asforeground or background.

In this embodiment, the acquired image on which face detection/trackingwas performed is thus compared with the second image of the scene todetermine 960 an average reflective characteristic k for each object inthe aligned images according to the following equation:

${{Avg}( \frac{L_{Flash} - L_{{Non}\text{-}{Flash}}}{L_{{Non}\text{-}{Flash}}} )} = k$

where L_(Flash) is the luminance of an object in the flash image andL_(Non-Flash) is the luminance of the corresponding object in thenon-flash image. If the value of k>0, the object is reflective and ifk<0, the object is not reflective, which situations may occur due tointerference or noise.

For each unlabeled object, i.e. objects which do not intersect or liewithin the region 82, having the same color combination as an identifiedforeground object, its average reflective characteristic k is comparedwith a threshold value k_(th) derived from that of the associatedforeground object, step 970. So for example, in FIG. 8 a threshold valuek_(th) is calculated as 70% of the average reflective characteristic kfor each object A . . . D, and a reflective characteristic k iscalculated for each unlabeled object. The reflective characteristic kfor each unlabeled object is compared with the threshold value k_(th) ofwhichever one of objects A to D corresponds in color to the unlabeledobject.

Thus, in the present embodiment, if the unlabeled object has an averagereflective characteristic k of greater than approximately 70% of theassociated foreground object, it is identified as a foreground object,980. Otherwise, it is identified as a background object, 990.

In the case where an unlabeled object comprises pixels having a colorcombination that does not correspond to the color combination of any ofthe identified foreground objects, the threshold value k_(th) forobjects of that color may be estimated as a function of the reflectivecharacteristic(s) of identified foreground objects, e.g. objects A . . .D, having the most similar color combinations.

In the embodiments of FIGS. 7 and 9 all of an object or sub-objectintersecting the section if considered to be a foreground object. In avariation of these embodiments, foreground objects intersecting theregion 82 identified in steps 940/950 further subjected to furtherluminance analysis based on their average reflective characteristic, k.

Thus, a sub-region of each intersecting object wholly within the region82 is confirmed as a foreground object. Now a reflective characteristicis calculated for each pixel of the object lying outside the region 82.Growing out from the region 82, object pixels neighboring the sub-regionare compared pixel-by pixel with the reflective characteristic of theobject sub-region with the region 82. Again, where a pixel value isabove a threshold proportion of the reflective characteristic k, say70%, it is confirmed as being a foreground pixel. The sub-region istherefore grown either until all pixels of the object are confirmed asforeground or until all pixels neighbouring the growing sub-region areclassed as background. Smoothing and hole filling within the grownregion may then follow before the foreground/background map isfinalized.

The invention claimed is:
 1. A method comprising: using aprocessor-based digital image acquisition device that includes aprocessor programmed by processor-readable code embedded that is storedwithin one or more digital storage media to perform the steps of:providing a first map comprising one or more regions that are within amain digital image of a scene, wherein each of the one or more regionshas one or more pixels with a common characteristic; providing an objectprofile corresponding to a region of interest of said main digitalimage, wherein the region of interest includes an object candidate;wherein said object profile comprises an object portion and at least asecond portion that is adjacent to said object portion and has a knownassociation and orientation relative to the object portion; determiningthat the object candidate within the region of interest matches saidobject portion of the object profile; determining an orientation of theregion of interest within said main digital image; comparing said regionof interest with said object profile to determine whether said region ofinterest matches said object profile; and designating said region ofinterest as a foreground or background region based on said comparing.2. The method of claim 1 wherein providing said first map comprisesdefining each of said one or more regions of said main digital image asa foreground or background region and wherein said one or more regionscomprise at least one region defined as a foreground region.
 3. Themethod of claim 1 wherein the providing of said first map is based on acomparison of two or more images approximately of said scene. 4.A-method as claimed in The method of claim 1 wherein the providing ofsaid first map comprises: providing two or more images each of differentfocus and nominally of said scene; calculating from said images a depthof focus map indicating pixels of said main digital image as either aforeground region or a background region; blurring said focus map;thresholding said blurred map to an intermediate focus map comprisingindicating regions within said intermediate focus map as either aforeground region or a background region; and filling the regions withinsaid intermediate focus map to provide said first map.
 5. The method ofclaim 1, wherein the providing of said first map comprises: providingtwo or more images each of different focus and nominally of said scene;and comparing high frequency coefficients of corresponding regions insaid images to determine whether said one or more regions comprise aforeground or background region.
 6. The method of claim 1 wherein theproviding of said first map comprises: providing two or more images atdifferent exposure levels approximately of said scene; and comparingluminance levels of corresponding regions in said two or more images todetermine whether said one or more regions are foreground or backgroundregions.
 7. The method of claim 1 wherein said providing a the first mapcomprises: analyzing at least one region of the main digital image in acolor space to determine a color distribution within said at least oneregion, said color distribution having a plurality of distinct colorpeaks; and segmenting said at least one region based on proximity of apixel's color to said plurality of distinct color peaks.
 8. The methodof claim 1 wherein said main digital image is one of a stream of imagesand the method comprises: determining at least one region of said mainimage that includes an object; detecting the object in at least oneimage of said stream acquired prior to said main digital image; andtracking said object through said stream of images to determine saidobject in said main digital image.
 9. An apparatus for providingimproved separation between foreground and background regions in adigital image of a scene, comprising a processor and one or moreprocessor-readable media that are configured to program the processor toperform a method comprising: acquiring a main digital image; providing afirst map comprising one or more regions that are within the maindigital image of a scene, wherein each of the one or more regions hasone or more pixels with a common characteristic; providing an objectprofile corresponding to a region of interest of said main digitalimage, wherein the region of interest includes an object candidate;wherein said object profile comprises an object portion and at least asecond portion that is adjacent to said object portion and has a knownassociation and orientation relative to the object portion; determiningthat the object candidate within the region of interest matches saidobject portion of the object profile; determining an orientation of theregion of interest within said main digital image; comparing said regionof interest with said object profile to determine whether said region ofinterest matches said object profile; and designating said region ofinterest as a foreground or background region based on said comparing.10. The apparatus of claim 9 wherein providing said first map comprisesdefining each of said one or more regions of said main digital image asa foreground or background region and wherein said one or more regionscomprise at least one region defined as a foreground region.
 11. Theapparatus of claim 9 wherein the providing of said first map is based ona comparison of two or more images approximately of said scene.
 12. Theapparatus of claim 9 wherein the providing of said first map comprises:providing two or more images each of different focus and nominally ofsaid scene; calculating from said images a depth of focus map indicatingpixels of said main digital image as either a foreground region or abackground region; blurring said focus map; thresholding said blurredmap to an intermediate focus map comprising indicating regions withinsaid intermediate focus map as either a foreground region or abackground region; and filling the regions within said intermediatefocus map to provide said first map.
 13. The apparatus of claim 9,wherein the providing of said first map comprises: providing two or moreimages each of different focus and nominally of said scene; andcomparing high frequency coefficients of corresponding regions in saidimages to determine whether said one or more regions comprise aforeground or background region.
 14. The apparatus of claim 9 whereinthe providing of said first map comprises: providing two or more imagesat different exposure levels approximately of said scene; and comparingluminance levels of corresponding regions in said two or more images todetermine whether said one or more regions are foreground or backgroundregions.
 15. An apparatus as claimed in The apparatus of claim 9 whereinsaid providing a the first map comprises: analyzing at least one regionof the main digital image in a color space to determine a colordistribution within said at least one region, said color distributionhaving a plurality of distinct color peaks; and segmenting said at leastone region based on proximity of a pixel's color to said plurality ofdistinct color peaks.
 16. The apparatus of claim 9 wherein said maindigital image is one of a stream of images and the method comprises:determining at least one region of said main image that includes anobject; detecting the object in at least one image of said streamacquired prior to said main digital image; and tracking said objectthrough said stream of images to determine said object in said maindigital image.
 17. One or more non-transitory processor-readable mediahaving code stored therein, wherein the code is configured to programone or more processors to perform a method comprising: providing a firstmap comprising one or more regions within a main digital image of ascene, wherein each of the one or more regions has one or more pixelswith a common characteristic; providing an object profile correspondingto a region of interest of said main digital image, wherein the regionof interest includes an object candidate; wherein said object profilecomprises an object portion and at least a second portion that isadjacent to said object portion and has a known association andorientation relative to the object portion; determining that the objectcandidate within the region of interest matches said object portion ofthe object profile; determining an orientation of the region of interestwithin said main digital image; comparing said region of interest withsaid object profile to determine whether said region of interest matchessaid object profile; and designating said region of interest as aforeground or background region based on said comparing.
 18. Thenon-transitory processor-readable media of claim 17 wherein providingsaid first map comprises defining each of said one or more regions ofsaid main digital image as a foreground or background region and whereinsaid one or more regions comprise at least one region defined as aforeground region.
 19. The non-transitory processor-readable media ofclaim 17 wherein the providing of said first map is based on acomparison of two or more images approximately of said scene.
 20. Thenon-transitory processor-readable media of claim 17 wherein theproviding of said first map comprises: providing two or more images eachof different focus and nominally of said scene; calculating from saidimages a depth of focus map indicating pixels of said main digital imageas either a foreground region or a background region; blurring saidfocus map; thresholding said blurred map to an intermediate focus mapcomprising indicating regions within said intermediate focus map aseither a foreground region or a background region; and filling theregions within said intermediate focus map to provide said first map.21. The non-transitory processor-readable media of claim 17, wherein theproviding of said first map comprises: providing two or more images eachof different focus and nominally of said scene; and comparing highfrequency coefficients of corresponding regions in said images todetermine whether said one or more regions comprise a foreground orbackground region.
 22. The non-transitory processor-readable media ofclaim 17 wherein the providing of said first map comprises: providingtwo or more images at different exposure levels approximately of saidscene; and comparing luminance levels of corresponding regions in saidtwo or more images to determine whether said one or more regions areforeground or background regions.
 23. The non-transitoryprocessor-readable media of claim 17 wherein said providing the firstmap comprises: analyzing at least one region of the main digital imagein a color space to determine a color distribution within said at leastone region, said color distribution having a plurality of distinct colorpeaks; and segmenting said at least one region based on proximity of apixel's color to said plurality of distinct color peaks.
 24. Thenon-transitory processor-readable media of claim 17 wherein said mainimage is one of a stream of images and the method comprises: determiningat least one region of said main image that includes an object;detecting the object in at least one image of said stream acquired priorto said main digital image; and tracking said object through said streamof images to determine said object in said main digital image.